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During fiscal year 1977^ training workshops 6n the 
new ESEA Title I eval,uation and reporting system were conducted for 
State level program administrators ai>d evaluators. In this report,, 
the workshop activitiesi are documented. The purpose:?^ and development 
of ^;-he evaluation system and the technical assistance efforts are 
summs^rized, "as an introduction to the program. In the remainder of 
the report, the content, materials and proce'dures of the workshops 
are focused upon. Their role in the overall teaching strategy is 
discussed. Also described are the revisrons of training materials, 
injpluding. the evaluation forms and instructioDS themselves, a6 well 
as 'techBical papers on subjects such as testing of Title I students 
and selection of students! Some of the complexities of running an 
intensive training program on a nationwide scale are outlined, 
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I. INTRODUCTION 

*• , * * ■» ♦ 

Dicing Fiscal Year 1977, KMC Research , Corporation, 'under conVract 

with the -U.S. Office of Education, Of fice of Planning, Budgeting, and ' • 

E>>aluation,^^nducted training workshops on the new ESEA Title I eymlu- 

* ation and reporting system .for State-level ^rogr^ administrators aneL ^^ 

-evaluators . This report docXmetits those activities that were carried 

out by, KMC ResBarch Corporation under ^cfntract 360-76-0316, ESEA Title 

* 

I Evaluation Workshops • - * < 

Initial development of the Title I evaluation and reporting^ system 
was begun by RMC in 1974, also under contract Vitb USOE.' In the follow-^ 
ing year further developmental work jJropeeded, together with an effort 
^ to present tlxe fundamental ieatures of the system to administrative and 
evaluation personnel in 'each of the States^. This effort helped to iden- 
tify needed revisions in the system, as well as to provide an estimate 
of the nature «and scope of the technical assistance that would be required 
for its implementation. One the resulting recommendations was that 
training assistance be provided directly to State educational agencies . 
(SEAs) through workshops. . * . \ , 

Following a brief summary af'^the purposes and development of the 
evaluation system arid of the technical assistance efforts,, *the^ remainder 
of this report will focus upon tlie"^ content, materieds, and procedures 
>of the workshops, ancTupon^ their role in the overall training strategy. 
Final revisions of the system dot'cUmefttatdon will also be described. 

Background ^ ^ * 

The -Elementary and -Secondary Education Act of,i.965 contains pro- 
visions for federal aid^to education. Its Largest and probably best 
known component, J^tle. I author iz^afinancial assistance to local edu- 
cational agencies (LEAs) ^that have^coribentrations of economically dis-, ; 
advantaged children.- Title I funds are tb provide special -educational' 
programs for educationally disadvantaged children in those sK^s. 



ERiC 



1 

7 



Federal evaluation requirements > Title I funds are allocated to 
SEAs for distribution to their eligible LEAs • It is the responsibility 
of the LEAs to design, carry out, and evaluate the effectiveness of 
their own Title I projects. 'Their evaluation results must be reported 
annually .to their SEAs. The SEAs are required to summarize local eval- 
uation data and submit annual State reports to USOE, which in turn re-' 
ports to Congress. 

In the past, a number of attempts have been tiaade to analyze and 
synthesize the data in these State evaluation reports, in order to form 
a picture of the nationwide impact of Title I. (See Wargo, Tallmadge, 
flichaels, Lipe, & Morris, 1972; Gamel, Tallmadge, Wood, & Binkley, 1975.) 
Unfortunately, the conclusion has ^always been that the State reports, 
takep as a whole, do not provide the information needed to assess nation- 
wide program effectiveness. There is simply too little uniformity in 
th'e type, format, and quality of information reported at each level. 
This predicament led to a^decision to* try to improve the quality of 

, local "evaluations, and to standardize reporting practices . Section 
151 of ESEA Title I, as added by the Education AmendmWts of 1974, laid 
out the ground rules for the new reporting approach; in particular, 

^paragraphs (d) through (f) specified the following: * ' . 

"Sec. 151. (d) The Commissioner shall provide to State 
educational agencies*, models for evaluations of all programs 
conducted, under this title, for their use in carrying out .their 
functions und^r section 143(a), which shalT include uniform 
procedures and criteria to be utilized by local educational . 
agencies-, as well as by the ^tate a&ency in the evaluation of 
such programs. 

"(e) The Commissioner shall provide such technical and 
other assistance as may be necessary to State educational 
agencies to enabla them to assist. local educational* agencies 
' in the developmen^arfd application of a systematic evaluation 
of. programs In accordance with the models developed- by the 
Commissioner. r ♦ 

"(f) The models developed by the Commissioner shall 
specify objective criteria which shall be utilized in the 
-evaluation of all programs and shall outline .techniques 
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(such as longitudinal studies of children involved in such ^ 
programs) and methodology (such as the use'sgf tests which, 
yield' comparable results) for produding*^ dataVhich are com- 
• parable on a statewide and nationwide 'basis .V ^* 

Development of the system * \jn4er contract to USOJT, RMC 'Research 

HP ■ 

Corporation undertook the development of prototype evaluation models as, 

specified in Section 151.. To determine the kinds of information, re- 
quired by the Federal government, staff members met with educational 
policy makers in Congress and the Department af— Health, Education and 
Welfare (HEW). Also, to determine the characteristics of Title I feval- 
uatitSns up to that date, they analyzed State reports for the years 1970 . 
to 1974.^ ''^ ' . . 

Based on the information gathered ' from these sources^ preliminary' • 
objectives for "the new system were developed. Reactions tb the initial 
development were solicited' in phone interviews With .the- State Title I . 
Coordinators'. Inputs were aflso received* from U^OE representatives, 
and the plans were revised on the basis of the State and USOE^sugges- 
tions. * ' 

A' reporting system was then develope'd during tbe winter" and spring 
of 1975 that 'waS' designed to maintain maximum autonomy of -local evaluaV 
tion efforts, while still providing data that wou^^be comjparable and 
aggregatdbie on a statewide and nationwide basis. The flow of informa-* 
tion from LEAs to SEAs woTi-ld continue. Project evaluators would be| 
able to use normed or rion-normed* te^ts of their choice, and to select 
from three basic evaluation models for assessing cognitive -achievement 
gains w 'These three were a norm-referenced model, a control group model, 
and a special regression mo4el . A new metric, the Normal Curve Equiv- 
alent (NCE) , was also developed that allows^'gains on all mjeasures ^to 



- it was later learned that an identical scale, cal.led the Stdndile 
Scale, had been dev^eloped by Frederick B. -Davis-,^ then of the .Educational 
Records Bureau. Unfortunately, it did not receive much publicity and 
disappeared from the professional literature. 



be^ expressed in (assumedly) equal- interval units derived £ro^ national * 
percentile ranks • The three evaluation models and the NCE score are 
described in greater detail in Chapter II of this report. 

Once the initi'al version of the system had been developed, reactions 
and' suggestions were again solicited frpm SEA evaluators and Title I ^ 
personnel, this time in nine pilot S<Ates, which were visited and pre- 
sented with the system in 19'75. Their confments contributed to a revised 
version *of the system, described in Gamel.et al., 1975 • 

In 1976, RMC conducted a project to review the methodology of the 
evalpation models, visit ^11 SEAs and a sample of LEAs in each State to 
discuss the system, dnd estimate the .resources that would be required 
to implement^ the system* ^ / 

The site visits to the States and their LEAs provided a great deal" 
.^of information about current evaluation pracjtices, reactions to the 
jnodels, Bnd further revisions nee^ded in the system. All 57 SEAs were 
vVsited, and the system was presented to "personnel from over 400 local 
distr icts , . or an'average of seven districts per State. The majority 
of site visits produced positive-; reactions to the system, but some 
SEA and LEA personnel were basically indifferent, and some were highly 
negatiy/. A number of SEA6, parti-cularly those that believed they al- . 
ready had valid evaluat,ion practices, expressed , some resentment toward 
a federally ^mandated , untested' system.^ 

r 

Many valuable comments were received from State and local persbri- 
nel during the site visits, and these contributed* to yet anothet round 
of revisions in the system and the forms and instructions that had been, 
developed for reporting data. It was this revised version of the system 
ihat was presented to SEA personnel' at the workshops carried out during 
th^ project de8crlbe4 in this' report. 
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II. BRIEF DESCRIPTION OF THE SYSTEM 

. A major goal in developing the Title I evaluation and reporting 
system was ° to provide meaningful, comparable information on all Title I 
projects across t\^e nation. The system was also intended to be flexible, 
to offer State and local evaluators conside'rable freedom in -choosing 
different 'evaluation designs and tests. By incorporating a common metric 
for expressing the results from the different evaluation models and dif- 
ferent tests, the system could at th^ same ^time allow the aggregation 
of impact data at the school, district. State, and Federal levels. 

Definition of a Title I project . As the system was |ieveloped and 
.refined, it became increasingly clear ths^t the project, the ,moi'ecular 
«nit around whicli data collection wais^ structured , had to be explicitly 
and uniformly defined. The de'finition of "project" that was eventually 
adoptee} reads as follows: . ^ 

, A project is defined as an instructional treatment with 
objectives, methods, materials, pers^mnel, and activities 
that' are uniform for all those it. serves. A project may 
exist in one or more schools within a district, or even 
i^ several districts. Many Title I projects are. clearly 
defined by their' application for Ending.. In case^ where ^ 
• , a" single application results in the funding of several 
qualitatively different instructional treatments at dif- 
ferent sites, t^he teifm project is reserved for the tndi- ' 
vidual tr^at^nents and not for tjie funded composite. 

Whatever definition^ of "project" schools or districts may currently 
use for administrative or accounting purposes, all must adhere to this 
definition when collecting and reporting data in the Title I system. 
Then, for each -project , certain basic descriptive informaV:ion is to be 
provided. ^ ^ 

TVpes of Information Collected \ 

The information 'collected in the. System fal^ls into six general 
categories: 
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• Student partic ipation---how many student^ are served 

at e^ch grade levels by public and non-public schools, 
and how^many sXudent-s are included in different 
subject-matter or ^rvace areas • 

V ."^ 

• Parent involvement — participation in parent, advisory 
councils and types of 'activities carried out. 

• .Personnel — number and types of*^^eJ^sonnel' employed J)y 

Title I projects. 

Training — areas and' amounts jo f training received by' 
Title I project personnel. / ^ ' - 

. ■ " ■ - t • • 

• Cost — overall cost of each Tattle I project. 

• Impact — achievement gains produced^ by each Title t 
project (^at present measured only in reading and 



mathematics areasO 



This information is collected and passed up through' v^^^us adminis- 
trative reporting levels for use^in the State report. 'The l^e of inter- 
mediate reporting levels to^e used ±9 a matter of local decision, <3ex- 
^plained more fully in the section below entitled. "Forms ,and Instructions. 
The forms and instructions were developed for a prototype reporting se- 
quence using three administrative levels: ^the building, the LEA, and the 
SEA. At. each level, the information from the lower levels is aggregated; 
however, in the^process of aggregati^igs project information is not lost. 
Instead , 'summary inform'ation on each project is produced at each level. 
At the LEA level, for instance, there is a summary of each project's 
gain and. its characteristics. The system will always permit g'ains to" 
be traced back to an individual project, thereby enabling" apparently ex- 
emplary projects to be studied*and, if appropriate, ^iissemin^ted . 
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The Evaluation. Model^ 



The .three evaluation mbdels use a common definition of the- treat- * 
ent effect in measuring jt:lie cognitive iini>act of Title I projects. The 
gain attributed to the treatment is defined as the difference between ' 
the\ treatment group'' s performance on. a post-treatment" test and an esti- 
mate' of* what .per formanc"^ on' th^ same 'test w.ould have been had the group, 
not received the treatment. In other words/ .the project's impact is 
the o bserved post-treatment performance minus .^the expected no-treatment 
performance. .This relationship oaii be expressed as follows: 
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Treatment *Ef feet = 



Observed 
Post-Treatment 
Performance ♦ 



Expected 
No-Tjreatment 
Performance 
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The observed post-treatment petformance is always the mean or median 
posttest score of the treatment group. The no- treatment expectation is 
derived through implementation gf one of the evaluation models. The 
norm-referenced design^en^^rates this no-treatment expectation from norms 
tables. The assumption is that, without the special .proj ect^ the^treat- 

-ment group would maintain its percentile status relative to a national 
or local norm group from pretest to -posttest. ,The control group model 
uses the posttest (or adjusted posttas^:) scores of a control group as 
the no-treatment expectation^ ^In the special regression model , the no-- ^ 
treatment expectation i^ derived by entering the treatment grou^^'s mean 
pretest score in the comparison group's post-on-pr etest r^^gre^ion equa- 

-tion. ' ' ^ ^ ^ • ' 

The following three sections provide overviews of each-'iuodel and 
o.utline"^ the .basic assumptions underlying them. 

Model A; Norm-Referenced Design . Model A can be implemented -^with 
either normed tests (Model Al) or non-rnormed tests (Model A2). Model j 
Al requires that the treatment group be'both pre- and postte&teiL with 
a nationally or locally normed achievement test. 
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. When tests with national norms are used , .the no-treatmefit expec- 

tation is found by detemjining the percentile status of the treatment 

group at pretest time. It .is^assumed that, without the Title I treat- 

inrent,,the status of the group a>t po^ttest time woulq be the same as it 
» 

was at pretest time.; Thus the group's pretest percentile becomes the^ 
exp^cced no-treatmetit posttest percentile. The observed post-treatment 
percentile rank is' that whicll^ corresponds to the group's mean posttest 
score. If the group's pos^ttest^ status is higher' than the no-treatment 
*expe.ctat ion (their percentile at pretest time^) ,^ then the gain is attrib- 
uted to. the' children' s partic ipatiori In the Title I project. 



When tests w^th local but no national norms are used, it xs necces- 
sary to administer a test with national • norms as well. The no-treatment 
expectation is derived from the local norms, following the same procedure 
just described. Data from the nationally normed test are needed only to 
convert the measured gain into NCE units. (NCE conversions are disqussed 
in a later Section of this- chapter .) 

\ • ■ ■ 

The norm- referenced model may also bfe implemented lising non-normed 
tests (Models A2). A nationally normed test, how^vet , must be given either 
at protest oc at posttest time in addition to the non-normed test. The 
model involves an equating o€ the scores on the normed and non-normed 
tests so that tl>G norms may be used to derive the observed posttest per- 
formance. In essence, the procedure is as follows (assuming that the 
norjned test was administered at pretest time) . The median pretest stan- 
dard score on the normed test ife determined. The pretest percentile cor- 
responding to this score is then read out of the pretest norms' table. 
This percentile constitutes the no- treatment expectation. The normed 
and non-normed tests are then equated. This enables the median pQSttest 
score on the non-normed test' ^^p^ doiwerted t^»lts normed test counter- 
part . This- figure, in' turn, is converted to a percentile using the post- 

> 

test norms table. X^e percentile derived in this manner is the observed 
post-treatment performance indicator. 

In the norm-ref ereoced mo^el, all pre- sand posttesting (normed and 

- ' s 

non-normed) must be done at times that are close to empii^lcal normative 
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da*ta points for the normed test * , (For Model A2 this means that the non- 
normed test must be given near the dates when normative data were col- 
lected for the normed test.) 

Mo^el B: Control Group Design > Model B is a control group design 
that requires that either a normed or non-normed test ^e given to treat- 
ment and control groups at pretest and posttest times. If a non-normed 
test is administered pre and post, a nationally normed test must also be 
given sometime"j?ScIrig the school Tear to the treatment graup. These 
normed test scores are used only to convert the measure .of gain into NCEs» 

Between pre- and posttesting, the educational experiences of chil- 
dren in the control group should be similar to those of the treatment 
group children with the single except ion' that the control group \ioes not 
participate in the Title I project. The control group's posttest per- 
centile (Model Bl) or mean raw score (Model B2) is t^ie . no-treatment ex- 
pectation. The treatment group's percentile or mean raw score on the 
posttest forms the observed post-treatment performance. When the treat- 
ment group's performance is superior to the control group's, it can be 
assumed that the project was effective. 

The pretest scores of the groups are used only to verify their pre- 
treatment comparability, or to quantify the initial difference' between 
the groups. Two statistical techniques are offered to adjust the post- 
test score difference for whatever pretest inequality may- exist • Where 
groups were formed by random assignment of children drawn from a single 
population or where two pre-existing groups are enough alike\to be con- 
sidered Random samples from a single population, the covariancXmethod 
of adjustment is used • Where small systematic differences- are ^nown or 
assumed to exi^t between treatment and control groups (in other words, 
when the groups are best^considered samples^ from different populations)', 
the princgLpal-axis oi standardized-gain-score method of adjustment is 
used. • ^ ' ^ 

The accuracy of the measure of project impact depends on thd\^ suit- 
ability of the control grou^.^ Ideally,^ the treatment and control groups 
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should/ be equivalent on all educationally* relevant dimeg^sions such ap 

socioeiionomic status, race, sex, etc» Small 'between-grouped if ferencfs 

can be adequately handled through the appropriate statistical adjustment 

# 

procedure • Large differences are likely to invalidate the models 

t 

The model permits .the use of different tests ov different, levels of 
the same test for pretest and posttest, and testing times need not co^n<^ f 
cide with norming dates • 

Model C; Special Regression Design ♦ Model C is a regression-based 
evaluation design. Apy, instrument , yating, or composite measure that 
correlates highly with the postt^est^may be used as a pretest. The post- 
test maybe either ^a normed (Model CI) or non-normed (Model C2) test. 
If a non-normed te^t is u^ed, -a normed test must also be given to the 
treatment group sometime during the school year. The model requires that 
the selection of the treatment group be based exclusively on the pretest 
measure (which may, however, be a composite of test scores, teacher rat- 
ings, etcx) • , All pupils scoring above the cutoff score must be*assigned 
to the coiiparison group while those' scoring' below form the treatment group. 
All pupils ia both the treatment and comparison groups must be pre- and 
posttested. ^ ^ 

Post-on-pretest regression lines are calculated separately for the 
treatment and comparison groups. The treatment group's regression line 
represents the observed mean posttest performance corresponding to v^ri^ 

rs pretest scores. The comparison group's regression line, when pro-"^ 

j'ected across the cutoff score, provides no-treatment estimates for the 

same pretest scores. (Illustrations of this are presented in Chapter VI 

2 

*of the Title I evaluation and reporting system User's Guide . ) 

The treatment effect is defined as the distance between the regres- 
sion ^nes and is. measured separately at two points: at the treatment 
group's mean pretest score and at the cutoff score. For both measures 



2 

Copies, available from Office of Planning, Budgeting, and Evaluation, 
U.S. Office of Education, 400 Maryland Avenue, S«W. Washington, D.C. 202O2, 
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.group's meawpretest score and at the cutoff* scqre^* For both measures - 

it is assumed that the pfoject had a positive-'^pact Ji^ the observed 

score is higher than the expected score** However^ a. substantial differ- 

ence between the two measures may signal a spurious apparent gain result- 

♦ 

ing from test ceiling or floor effects* ^ , 

There are few constraints concerning test selection* Model C does 
not require the same test for pretest and posttest ot'thatcthe tests , be 
administered at norming times* It is essential, however*, that the pre- 
test and posttest scoter be highly cori^lated and that the instruments 
not be so difficult or so easy that test floors or ceilings are encountered 
The model assumes a linear relationship between the pretest measure and 
posttest scores; where floor and ceiling effects exist, the pretest/post- 
test relationship will no loiter be linear* Finally, in Model C, scores 
on the pretest measure must be used as the sole bas\s for selecting ,the^ ■ 
Title. I project participants* 

Normal Curve Equivalents . * 
<c 

Normal Curve ^Equivalents (NCEsX^are normalized standard scores with^ 
a mean of 50 and a standard deviation of 21*06* The scale is normalized 
because it is assumed that the characteristic being measured (e*g*, read- 
ing achievement) is normally distributed in the population* To, the ex- 
tent that this assumption is valid, a normalized scale will be an equaj- 
^ interval scale — that is, the length of the interval between any two ad- 
jacent scores on the scale is eqiial to the interval between every other 
adjacent pair of scores* ' ^ - 7^ 

The fact that the scale is standardized means, simply that the scored 

have been linearly transformed to give tljem a destined .^ean and standard 

3 / ^^A- 

deviation* The values of 50 and 21*06 were chosen to. r^elate NCEs to the 



3 

The NCE standard deviation of 21*06 was derived -by dividing the 
distance from the mean to the $9th percentile (99 - 50 = 49 percentile 
points) by the/ same distance measured ,in terms of normal curve standard 
deviation units (2*3267 -'O = 2*3267)* The result (49 ^ 2*3267 = 21*060) 
yields a scale t|iat includes exactly 98% of the population betweian val- 
ues or 1 andHj^. 



11 



17 



. petite entileps^ ale, thereby lending intrinsic meaning to them through the 
association. The exact relationship is shown in Figure 1. 

The two scales match at values of 1, 50, and 99 but, while the NCE 
pcale points are equally spaced,*' percentiles are widely spaced at extremes 
of the distribution and tightly clustered near the center. -It dan be seen 
in^ Figure 1, for example, that the distance between NCEs of 10 and 20 is 
"the same'as the, distance between NCEs of 50 and 60. In contrast, the dis- 
tance between percentiles o^ 10 and 20 is nearly twice that between per- 
centiles of 50 and\o.* 

Because pert:entiles are not all the same size — that is«, their incre- 
ment^ represent different amounts of change in achievement — they should 
not be used in arithmetic computations. The NCE metric is assumed to be 
an eaual- interval scale, however. Because of this characteristic, it is 
legij/imate to add, subtract, and average NCEs. - ^ 

Figure 1 also illustrates the relationship between NCEs and stan^nes. - 
Whereas NCEs divide th^ distance under the normal curve into 99 equal * 
^units, stanines divide the same distance into 9 equal-sized units. Thus, 
there are 11 NCEs.j&or fevery one stanine. NCEs closely resemble stanines 
and botji scales ar;s equal- interval ; however,. NCEs are a more finely grained 
metric. 

Measuring gains in NCEs ^with nationally riormed tests . From each 
model, it is possible to derive >a no- treatment expectation (via norma- 
tive data, control group data, or comparison group data) which can**be 
compared against* the -Title I children s observed post-treatment perform- 
ance to yield a measure of project impact. When nationally normed tests 
are used pre and post, observed and expected posttest scores are typi- 
cally expressed in terms of percentile ranks with respect to a national 
norm group or in terms of the NCE counterparts of these percentiles. 

System users are provided with a percentile- to-NCE conversion table, 
which was derived from the table /(found in most statdstics books) of 
areas under the normal curve. Using this conversion table, observed and 
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Figure 1. ^ Area under the ^normal curve divided into NCEs, 
percentiles, staj^n^s, and standard deviation. 
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expect;ed pbsttest -percentiles , can be/convertecl directly to NCEs. the \ 
NCE gain is 'the d±ff erence , between the observed NCE*and the expected one. , 

Measuring gains In NCEs with'' non-normed tests , .Calculating ,^n NCE 
gain when non-normed te^ts have beeti ;adminiatered is. a sligh-tiy more com- 
plex procedure than when normed tests, are used. Without nationa]^ly repre- 
sentative nornf^tive data , ,it is generally not passible t-o obtain national 
percentile equivalents for the observed and expected postte^t'^ scores (Model 

is an exception to this general ru'le)^ . ^/Irf the* absence^ of percentiles, 
the measures of observed andL expected postjtest status cannot be translated 
into NCEs. It is possible, however, to*expres,s the gain pLn NCEs. The 
formula is . • . • 



T- ,E- = 



21*06 (Y - Y ) s 
^ t t^ n 

0 i s 
n nn . 



(2) 



where Y = observed mean posttest 'score of 'treatm^rit group pupils 

Y = expected mean posttest score 'of treatment group pupils 
under *no-treatment conditions 

s = standard deviation of treatment group scores on the 
^ normed test 



nn 



standard deviation of treatment group scores on the 
non-normed test 



0 = standard deviation of norm group scores on the nprmed test 



(The complete derivation of this formula is presented in the system 
User^s guide . ) ^ ^^^-ww 

Measuring gains in NCEs using tests with local norms . When tests 
are used that have only local and no national norm% the gain is calcu- 
lated using the procedure employed with non-normed tests. A nationally 
normed test must be given in order ,to derive an estimate^of a national 
sample's standard devia'tion on the locally normed test. The necessary 
values are inserted in Equation (2) and the NCE gain can then be calcu- 
lated • ' ' * 
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When tests are given that have both' local and national noras> a some- , 
what? different procedure is used* After the standard -score difference 
between observed and expected pqsttest performance is determined, It is 
then divided by the standard deviation of the nationally representative 
sample's test scores • In contrast to the situations in which non-normed 
tests are used, the national sample's standard deviation does not have 
to be estimated; ^instead it can be obtained from the^ technical m'anual for 
the test. After the standard scare gain is divided by the national stan- 
dard-deviation, itkis multiplied by 21.06 ;to obtain the NCE gain. • 

Forms and Instructions 

A system of reporting forms and instructionte was designed to facili- 
tate the aggregation orf information throjigh three administrative levels — 
the "building, the LEA, and the SEA. Data on each project; could be cpl- 
lected at the building level, and sent up to the LEA level, where they 
are aggregated to form LEA reports. . The' data- in LEA reports, could then 
be aggregated to produce the State^^fiports, /wftdch are then used at the 
Feder^^ level to gauge the impac t:\Tf j Title /I- nati^ * ^ 

Ifeing the forms provided,' personnel jLtyeSch of the three levels can 
collect, analyze, and pass on exactly theAnformation required in all six 

•reporting areas. For the five general yl^iformatlon areas, the forms 'are 
identical regardless of which modfcl is' used; for the impact data, differ- 

tent coior'-coded forms, appropriaoa to the different models, are provided. 
To aid users* in coapleting the firms, detailed instructions are given at 
each level for both the general and the impact (model-specific) informa- 
tio^i. The instructions take the user step by step through implementation 
of the appropriate model, using a method that requires a minimum of tech- 
nical evaluation expertises (Further description of the forms appears 
in the section on "Training Materials" in. Chapter III.)^^ 

Although^ the reporting forms are provided for system users' conveni- 
ence, alternative methods qf collecting and aggregating data are perfectly 
permissible.* Thus, some States may wish to aggregate and analyze data 
collected directly from the^ buildings, bypassing the LEA altogether.' Or, 
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a dlst^ic-t may carry out. directly the collection of distr±c-fe-^4d^ d*ata, 
elimlivating the need* fpr reporting from the individual buildings. Some 
States might have "Intermediate" or consolida'ted districts that would ^lay 
a role lii Title I*eval,uation and reporting. X^e sets of form? and in- > 
structions could be adapted to their role. The^building-LEA-SEA Hierarchy 
was chosen because it was common and yet could be easily collapsed or ex- 
panded upon to meet administrative configurations in various settings. 
Any data-collection procedure preferred by the schools, 'districts^ o^ 
States is acceptable within the system. ,The sole requirement is that at 

*^ -16 " \ • 9 

A** 

.the State level, all SEAs must^ report thfe ^same kinds of informatipn. 
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III. DESCRIPTION OF THE WORKSHOPS . 

, f 

Perspective on the Training Effort 

^ In order to help SEAs get acqukinted with the requirements of the 
^system and formulate sound evaluation plans, USOE set up a comprehensive 
training effort that was intended to give State personnel a basic un- 
derstanding o^ the models and the reporting forms, to prepare them to^ ^ 
train local evaluators, and to provide them with on-going technical as- 
sistance as they began to implement new evaluation procedures. 

Technical Assistance Centers . The on-going consultation ^services 
are provided by ten Technical Assistance Centers (TACs) , one in^each. 
HEW region. The T^C s^rv^s all five or six States in the region, offer- 
ing training sessions for local personnel and advice in connection with 
specific evaluation problems identified by each* State. Staff from the 
TACs mak6 periodic site visits, and offer technical assistance as re- 
quested. . . . 

\ They also ^have a role in the continuing examination and. refinement 
of the system. Each TAC was given certain methodological questions to 
be studied, and inputs frgm the actual experience of, districts beginning 
. to implement the system'would contribute to the analysis of these prob- 
lems. TAC di^ctors maintain periodic contact with each other and with 
USOE to slia^e/liiforraation, questions, and findings. ( 

' Newsletter . In order to provide a link Vetween U^OE "and all of the 
States as they began their implementation efforts, a newsletter was- 
started. Issued periodically by US.OE, it carries information about the 
progress of Federal regulations for Title I evalua'tio'n and other pro- 
j^^s^om important to ^tate and local personnel?^ From time to time its 
cpntents may include letters from SEA states with comments or ^qiXestions 
about the syfetem, as well as answers from USOE persbnnel. An occasional 
patents' column appears, with comments from Title I parents. LEA per- 
sonnel also, contributed feature columns dealing ^with evaluation proc^ 
dures or innovations in their settings. This newsletter is distributed 
to SEA and LEA>BXaffs and' other interested persons in every State."" 



Monograph^ series . USOE also intended that reference books or "mono- 
graphs" be available to provide background information, more detailed 
explanations of methodological matters, and further references about eval- 

.V 

uation problems. They were to be user-oriented and' practical in nature 
and were— distributed .through the mail-order procedures ol the Government 
Printing Ofefice as well^as through" USOE offices. Two appeared early: 
A Practical Guide to Measuring Project Impact qn^ Student Achievement: and 
Aj.Procexiural Guidfe for Validating Achievement Gains in Educatiorial Proj- 
ects • Others planned for later years were to deal with assessing test 
bias, using criterion-referenced tests, measuring the, child's affective ' 
development, and estimating project costs, . . ^ * 

Workshops • The first intensive training of State personnel took 

place in the workshops described in this report. During the sit^!|yisits 

conducted in 1975-^76, most of the ^itle I Coordinators and evaluation 

personnel from the nation's 57 State Departments of Education had attended 

^briefings at which vi&iting RMC personnel described the evaluation and 

reporting system, ^n addition; many became acquainted with the system 

through presentations, by USOE personnel and others, at Federal and re- 

^ ' <? ' ' 

gional meej:'lngs,\ Although a number 'of States and LEAs shortly decided 

to adoj>t the 'system, it became clear whenever implementation was tried 
that few^were. adequateLy prepared. When it was decided to'^opt the sys- 
tem. t>n a nationwide basis, it was evident that' a coordinated effort would 
be required to give SEA personnel a thorough familiarity with the system, 
and to enable them to train district- level staff members in their States, 
The purpose of the workshops was to provide this training in the most 
efficient way possible within a relatively, short period of time,* 

The workshops wei:e planned to take place over a period of about 
four montn^in the fall and early winteif of 1976,^ One workshop was to 
Jbe held in each of the 10 regional areas of DREW. In addition to three ^ 
days of intensive training, attendees would receive materials that could 
subsequently be used to train other .personnel within their States, 
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Workshop Participants 

It was originally planned that each workshop should be" attended by 
two Or three persons involved in Title" I evaluatipn in each State in the 
region. Various o.ther personnel also attended, however, which enhanced 
the diversity of ^ the interactions and increased £he challenge of provid- « 
ing for different interests anid areas of expertise within a single three- 
day workshop. On the average, there were about 33 people,^,^tending each 
workshop* 

Personnel from SEAs /' Staff members who represented State educational 
agencies at the workshops usually came from several different branches * 
of their departments. The State Title I Coordinator almost always at-, 
tended. Usually, one or more representatives of the State's evaluation 
department were also present* Occasionally, computer center staff or 
State assessment testinjg personnel also participated* The Bureau of In- 
dian Affairs, which had Title I pr oj ects' locatedMn many States, sent 
repjresentatives who were working in those States to the appropriate re- . 
gional workshops. In all, a total of 243 SEA personnel attended work- 
shops; the average numlJer attending from^'e^ch State was about four. A 
complete list of the SEA representatives who ^tended workshops is pre- 
sented in the Ap^ndix. 

Personnel firom- Technical' Assistance Centers . As a separate part 
of USOE's plan for providing training and consultation to State and 
local education agencies that were seeking to implement the new evalu- 
ation system, a Technical Assistance Center (TAC) had been set up in ^ 
each of the ten HEW regions.^ Each workshop was attended by staff mem- 
bers of the Technical Assistance^ Center for that region. Although fa- 
miliarity with the system was an important qualification for the staff 
.of those organizations that received contract^, to establish Tecfinical 
Assistance Centers, the workshops provided additional training for them, 
as Well as an oppoirtunity to pose questions, make suggestions, and con- 
sult with RMC st^f. Anotlfer important' purpose of their attending the 
workshops was to establish personal contact with the representatives of 
the educational agencies with whom they would be working. Through these 
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meetings they could find out what evaluation plans were being made in 
thB various States, what questions had^'STlsen^ and what kinds of training 
and consultation were needed • 

Federal representatives ♦ Every woi^kshop was also attended by at 
least one staffs member from USOE's Office of Planning, Budgeting, and . 
Evaluation (OPBE) and by-a. represent>ative from the Title I office. These 
Federal personnel provided workshop participants with information about 

V 

the background and development of the Title I evaluation and reporting 

\ . • ^ . 

system, interpretation of the legislation, anda^plans for th^ ^forthcoming 

regulations* They were also available to respond tOL^miriistrative ahd* 

policy-related questions throughout the three-day sessions, and to clar- 

ify the Federal role in implementation of the system. 

i 

Staff • . ^ J 

The workshops A^ere presented by an instructional team consisting o'f 
five raiC staff members who travele'd* to all ten workshop sites. All five 
were* thoroughly familiar with the system. Two had worked on its devetop- 
ment from the beginning; these two and one other had been involved in the 
previous year's site visits to -the. 57 SEAs. During these visits th^ had 
presented the system to 1-arge numbers of State and local -personnel , often 
conducting mini-workshops for small, or large gathelcings . The other two 
staff members had experience in /tonducting training workshops on the 
system. * , i 

Staff training . Since all the members of the instructional team 
were experienced in presenting the system"to new users, staff training 
focused on developing and refining the spec.iiic curriculum and technicfuefe, 
that would be used for the ten workshops . Various activities contr ibutec^. 
to^- the training of team members. Entirely new problem sets were devised - 
for .both ver^sions of each of^the ,three "model s . Creating these simulations 
of actual ^valuations using each model helped the staff to visualize the 
exact >steps that trainees would be going through and where they might have 
dif ^icult-ies . Similarly, waiting the technical papers provided ideas for 
effective ways of presenting various concepts. 
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Staff scheduling * Once playis for the workshop agenda were formulated 
responsibility for the particular sessions was divided up* among the staff 
members ^ An. attempt Was made, to apportion the workload evenly and tc^^i^or- 
vide variety in the schedule of presenters on each topic. There were sev-^ 
eral reasons for this vari&ty. Oi^e was simply to of fer* some changes of 
ace /or those ,at tending the workshopfe. Another was to ensure that each 
^*m\nember was trained to cover severarl important segments of the curric- 
ulum and .would be fully^repared. to take over another's- scheduled presen- 
tation if the need aro&e. Thus, for example, the overview of Model B on. 
the first morning was prey^nted by one person/che i^-deptl]! Model B ses- 
sion by a second, and the Model B overview* o^ videotape' By a third. 

*■ . ■ _ : . A / 

Scheduling and Logistics ' * , " , 

* ^ O ' ^ 

It was originally planned vthat a to tar of eleven woricshops would be 
^conducted>, one in each of thfe ten HEW Regions, aod a "^rel^iew" workshop 

presented to\Federal personnel in Washington)^ D.C. This first work- 
was intended, to acquaint inlferested' persons wit^i t1ie system, and to ; 
serve as a rehearsal for the staff conducting, th^ ten regional workshops. 
The Directors of each of the ten* Technical Assistance Centers were also' 
invited to attend thfe Washington whiffles hop and to comment on the content 
amiformat of the \rorUshops .' The idea was that appropriate revisions 
dould be made, based. on reactions to the rehearsal, belore the rest of 
the workshops began.' "^n • * 

Workshop dates ^and locations . A tentative schedule for the ten » ■ 
regional workshops' was originally designed to include all the worltshops 
within the period from mid-September to mi^==»»^ember . During^ the summer, 
letters \^pere/sent " to all S'tatfe Tit^e I Coordinators, and follbw-up tele- 
phone' calls were made to discuss the tentative dates of the wofk^op and 
the most appropriate persons ^iri the SEA to attend. y *^ 

Th6 preliminary ^hedule for the workshopl^ staff included three two- 
wejek trips, three one-week*^ trips , and one workshop to take place in the 
local area (San Francisco) . "It was designed to minimize the tline and 
expense requirecf for travel by the KMC , instructional 'team. The J:wo-week 
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trips allowed two workshops to be con3ucted back-to-back with a weekend 
between, thus* requiring only one trip each to the most distant regions, 
the northeast, southeast, and midwest • ^ 

The cities that served as workshop sites were chosen on. the basis* • 
of convenience. They were large centers within their respective HEW 
regions, easily accessible by major air carriers, and able to offer ^ 
appropriate accommodations for participants and locations for ♦ the meet- 
ings. All sessions were headquartered in h6tels that could provide 
sleeping rooms, eating accommodations, and conference facilities for the 
participants under a single roof. 

Through negotiations with the various States and with Federal offi- 
cials, t4ie tentative schedule for the workshops was altered>^to avoid con- 
flicts w;lth other commitments. Owing to a meeting^f Title I coordina- 
tors in Washington,, workshops originally planned in Denver and Seattle ' ^ 
had to be cancelled, and a single workshop in Portland was substituted. 
The number , of workshops was thus reduced from ten to nine, and the des- 
ignation of' which States were to attend each one was revised. In r'evis-- 
ing the^che'du^e,' RMC sJba'ff tried to allow all State^^to attend at the 
most logical and convenient location. By the *time^he schedule ^as fi- 
nalized, the dates had been rearranged so that the workshops actually took 
place from ^mid-October to mid-January. Except for the week of Thanksgiv- 
ing and* the Chpistmas-New Year period, one 'workshop was held every week 
beginning in November • A copy of the/f in^l^chedule, including the dates 
and location of each workshop and the nkmes of' the States attending , .ap- 
pears in Fig^'i^e 2. * " - - 

V Arrangements for participants . Travel arrangements and reservations 
for all persons attending the workshops were coordinated by RMC staff, 
who took direct responsibility for hotel reservations. Participants were 
responsible for scheduling their own travel arrangements to and from work- 
shop sites. A pre-workshop packet sent to each participant contained a 
map showing the conference site, reservation confirmations, an airline' 
ticket (coach), and voucher forms for recording expenses. After th€^ work- 
shops, these forms were mailed back to RMC. Charges^tor rooms and per 
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Figure 2. Schedule of workshop dates and locations. 
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diem costs, personal car mileage ,^'>^o lis, cabs, and necessary ground trans- 
portation-were then reimbursed throii^ the KMC, office. 

One other -item that had to be' prearranged with the hotels was the ^ 
provision of refreshments for the mid- morning^ and mid-afternoon breaks • 

-This" turned" out to be a fairly' large ex^p^nditure, but an important one» 
The sessions were long and intensive,' and the breaks were extremely use- 

-►ful in J.etting participants stretchy their legs and regroup tljeir thoughts. 
Equally important, they gave participants and workshop staff a chance to 
chat informally ^bout the presentations , or about participants' individ- 
ual concerns. ' ' 

Provision of materials . Another responsibility of the RMC staff was 
to arrange for -the reproduction of training materials, and "for shipping 
them to the workshop sites. The training materials included binders con- 
taining forms- and instructions, problem sets, technical papers, and the 
User'^s Guide to the system. (These materials are described in detail later 
in this chapter.) Prior to the workshops, the User^s Guide and technical 
^papers were written, and problem sets were devised for^oth versions of 
each of the three models. Six hundred copies of each item were printed, 
and 600 three-ring binders were ordered. A binder for each workshop p^arr 
ticipant was assembled by RMC staff, so that everyone would have his or'- 
her own personal reference source containing all of the training mater ials. 

The bitiders were to be distributed to participants on the first^* 
morning of each workshop, as they would be in constant use during the ^ 
following three days. Since the traveling workshop staff could not carry 
all of these materials along with them, arrangements had to be made ahead 
of time vf or shipping the binders to each site» At each hotel, the manager 
was asked *to receive'' the boxes of materials and to hold them until; the 
RMC *staff arrived. The binders were packed six or seven to a large box 
and sent by comn^cial parcel service to arrive a day or twp, before each 

workshop. ^Videotapes and overhead transparencies, also packed in boxes,' 

^ / 
were sometimes taken as airline baggage by workshop staff members • 

Unlike materials to be used by participanirs, materials essential to' 
the presentations were always carried by workslvap staff members. These * 
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Included one set of overhead transparencies, any necessary notes and ref- 
erences, and -an overhead projector. S^ce it was too fragile to be ship- 
<ped or sent as baggage, the projector was taken on board all flights as 
carry-on baggage. Having one's own overhead projector appeared to be far 
preferable to trying to rely on rental equipment. Screens were supposed 
to.be provided by the hotels, but in ^ver^ia^ eases were obtained only with 
some ^^^iTul^ once the sessions had begun. - / 

Workshop Curriculum . 

The workshop curriculum was designed with several considerations in 
mind. First of all, it had to provide program and* evaluation personnel 
from each State witH an understanding of the three models and the NGE met- 
ric. A basic grounding in these essential aspects of the system would 
be necessary for anyone who would have to take an active role in planning 
or providing administrative support for its implementation. Secondly, 
the training had to be tailored in prder to accommodate the different in- 
terests and priorities of those attending; some would be interested in 
the specific technical and evaluation issues, while others would be con- 
cerned almost exclusively with administrative and pop.icy-rel^ted matters 
pertaining to implementation of the system. More ciian just a basic under- 
standing of the requirements of the models would 'blfe essential for the 
'former group, but not necessarily justified for .diie latter. Finally, 
both groups* had to be equipped not only with information that they them-^ 
selves could use, but with the means to train ptJhers within their States 
as well. ' ■ / ' 

Planning the curriculum . These goals meajtt that the workshops had 
• to cover an immense amount of material in so^ae depth over a three-day 
period. The diverse needs and interests al^fd meant that^not all par- 
ticipants would need to learn the same material at the same level of 
sophistication. Therefore it was decided early that the agenda should 
make considerable Us^e of "streaming," with parallel sessiohs for the 
different groups, based upon their ne^<Js. ° V 
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, The preliminary curriculum plan called for a vrtiole-group session " 
on the first mornings „ This was to consist of some introductory remarks 
to acquaint participants with the legislative background and the Mstory 
of '^the system's development, and to give them an overview o^^the three 
evaluation models and the new NCE metric* The afternoon session was > to 
be divided into two groups • For evaluators there would be a session on 
the forms and instructions both those for reporting general information 
(participation, parent involvement, project cost, personnel ,%nd tra^ining) 
and those for recording impact data» As a result of this session, eval- 
uators should understand the form^ and be able to train others to fill 
them out correctly. At the same time, , administrators would attend a ses- 
sion focusing on the overall flow of information within the system, the 
responsibilities of people at each level, and the interpretation of data 
to various audiences^. 

The second day was again scheduled to begin with a whole-group ses- 
sion, designed to acquaint all participants ^th Model AI, which was 
likely to be- by far the most widely used model • Everyone woul-d also^be 
involved in smalJ.-group tutorial sessions on the Mo^del AI forms and in- 
structions, including a problem set that would simulate the actual pro- 
cess of completing a Model AI evaluation. In. the afternoon, the -entire 
group would listen to* an in-depth presentation of Models CI and C2, but ^ 
then'only evaluators would work through Model P problem sets, while ad- 
Bjinistr atots gathered in small-group tutorials or^nized around specific 
-^juestioris that might have arisen, such as how to verify proper implefaen- 
tat ion of the models, what to do about neg^ative findings, or what to 
ct from audits. 




The third day was originally planned ^to consist almost » entirely of 
parallel sessions. This time three'dif ferent groups would be grun in the 
morning, one for" evaluators interested ^n Model A2, one for evaluators 
interested in Models BI and B2, ancJ"one for administrators on policy 
issues that have been^i^aised?. frequently-, such as test bias, us^ of grarfe- 
equivalent scores, and howAo evatU^te^^E gains. The afternoon was to 
begin with a whole-group session to wrap up^iscellaneous technical issues 



of interest to both evalU^toi^ and administrators*, such as ^out-of-lev^s-l * 

testing, adequacy of published* norms, and possible use of local hoAns» 

During the second half of thes afternoon, parallel aessions would •'provide* 

* . • ' • 

(a) individual consulting to those evaluators and administrators who 

raised special questions, and (b) a general review o^ the hazards asso- 

K 

ciated with each model foi;, those who had no special problems to <iiscu'ss. 
. ^ ' * * 

Results of Washington preview r The trial-run .workshop held for Fed^ 

eral personnel in Washington served its ^intended purpose very well and 
pointed up a need for several revisions in the ct&riculimi. One thing ^that 
became clear was that the problem .sets were too long, the' Model Al. prob- 
lem set in particular. This problem set had been designed to include 
all possible variations thajt^. trainees^ight encounter in using^ Model Al« 
It contained simulated data from several different projects at several 
different grade levels, with some projects using national norms and one 
requiring the development ^and use of local norms. It simply could not 
be finished in. the time allotted. After the Washington workshop, a short- 
ened versidn wa,s devised that in^bluded fewer projects and fewer grade 
levels. Also, forms were to be completed only f or. th'e Building and LEA 
levels; aggregation of the data at the State level was deleted. 

The trial run also showed' that breaking the l^irge group into small 
tutorials for the problem set only served to guarantee that the groups 
would become increasingly farther apart from each other in completing 
each section of the simulation, • and that eventually one group would be 
finished and idle while others were hopelessly behind. For this Reason 
it was decided to walk the entire group through the simulation step. by 
step together, rather than spJ.itting them up into small^groups . Simi- 
larly, in the adm^istrator sessions,' it became clear that there were 
few policy issues raised that'were not of interest' to all of the admin-* 
istrators; splitting up intp small-group sessions simply produced re- 
dundancy. 

On the other hand, the trial-run workshop confirmed the value of di^- 
viding tlfe total group into two* parallel sessions, one for evaluators an'd 
one 'for administrators. Some of those attending, such as the Tec^tnix^^tl 
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Assistance \ Center personnel and those directly involved in Title I evalua- . 
tion9i,\were\ eager to discuss technical aspects of the models and to explore 
the ^pl^ataons of the system for their own particular evaluation plans. 
Those who w^i^e not technical specialises, however, were npt equipped to 
cope wxth these discussions and had little interest, in them. These were 
generalily administrators, whose first concern wa^ with .the practical and 
ppllcy-Velated aspects of the system, problems such as how the models would 
affect ^heir regular testing schedules, or the selection of Title I part^^^^, 
cipants,'^or the manner in which results would be reported to parents. The 
parallel ^sessions met the needs of both groups while preventing the members 
of * either^ on^ from becoming bored or frustrated. 

\\ ' > ' 

On tfHe v^ole, the Washington workshop reinforced the conviction that 

there was probably too much material to ber absorbed, even if it could be 

adequately presented, in only three days. As a result, emphasis was placed 

on i^einforcing *the basic concepts of the three models and giving partici- 

pantW a "f^el" fpr the technical and practical requirements of the system, 

rather than on mastery of the problem sets and in-depth understanding of 

specific methodological and policy-related issues. 

The final agenda . Figure 3 presents a typical agenda from a pre- 
workshop packet fpr one of the later workshops. As the first few^ work- 
shops were compl^ced, the impressions formed after the Washington trial 
run were strengthened. By the time of the last few workshops, staff mem- 
bers had incorpbrated some further changes not reflected in the official 
agenda. The'se changes were based on participants' i^ppnses. For example, 
in order to get th^\essential material covered during the three-day per- 
iod, staff members* found it more effective to avoid additional splitting 

of the parallel evaliiator and administrator sessions into small groups. 
' \' 

Further consolidation\ of the third-day parallel sessions took place, with 

*'\ ' * 
all evaluators being given presentations on both Model A2 and^^odels Bl 

and B2, instead 6f having to choose between attending one or the other. 

They could then choose \to do a problem set for whichever of these models 

was of greater Interest and. practicality for them. Several staff members 

were available to ^rov^ide individual ,help while participants worked on 

the prpblem sets. . k 
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^5SEA TITLE I EVALUATION WORKSHOP 



AGENDA 




00 .a.m. - 12:13 p.m. 



Tuesday » January 25 ^ 
♦ % - 

General Session ' . 

.A Development of the Evaluation and 
- Reporting System. 



USOE Representative 

Introduction to the Syst 
the Evaluation Models. 



and 



mc Staff: Barbara Fagan, 
Oscar Roberts, Sarah Roberts, 
Kast TalTmadge, Christine Wood 



12:15 p.m. - 1:30 p.m. 



Lunch 



1:30 p.m. * 4:30 p.m. 
(Parallel Sessions ) 



Session A for Evaluators 

1:30-3:00 - Orientation to Training 

Ilaterials and General Forms and \ 
Instructions . 

RMC Staff: Sarah Roberts 

3:00-3:15 - Break 

3:15-4:30 - Impacc Forms and Instructions. 
RMC Staff: Christine Wood 

Session 'B for Administrators 

1:30-3:00 - Overview and Rationale of the 

Reporting System. Building, LEa^ and 
' SEA responsibilities. Definition of a 
project. ' 6 



RMC Staff: Barbara Fagan 
3:00-3:15 - Break 



ERIC 



3:15-4:30 - Preparing the State Report. 
Interpreting the data to legislators, 
school administrators, and parent groups. 



RMC Staff: Barbara Fagan • 
Figure 3* Typical workshop agenda. 
29 



35 



Agenda - continued 



Wednesday, January 26 



9:00 a.m. - 12:00 p.m. * General Session 

«^ .-^ . 9:00-10:15 - An in-depth look ^t 

{ - * Model Al. ^ 

• . * ^ RlIC Staff: Kast Tailiaadgfe 

/ 

10:15-10:30 - Break * 



10:30-12:00 - Simulation of a Model 
Al evaluation using both national 
and local noras. 

RlIC Staff:. All 



12:00 p.m. - i;15 p.iii^ Lunch 

1:15 p.m. - 2:45 p.m. General Session 

An in-depth look at Model C. . , 

Rl'IC Staff: Christine -Wood 

• L ■ ' • 

2:45 p.m. - 3:00 pvtn. Break 

I. 

3:00 p.m. - 4:30 p.m. ^ Session A for Evaluators 

(PaMllel Sessions) Simulation of a* Model C evaluation. 

\ ' . RliC Staff: All 

Session B for Administralg^s 

Small group tutorials on administrative 
issues. Audits, verification of proper 
model implementation, adequacy of criterion 
referenced tests. How to interpret and • 
what"" to do with negative^ findings. 

KMC Staff: All 
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Agenda' - concluded 



Thursday, January 27 



9f00 a.n, - .12:00 p.m. "^ Sessicfn A for Evaluators 

^(Parallel Sessions) An in-depth look at hodel A2 and" simulation 

* . r of a Modgl A2 evaluation, 

^ ' *' * . • ♦ ' . 

• ' RMC Staff: Sarah R,oberts i 

' i 

Session B for Evaluatbr'fe 

An in-depth look at M6del B and simulation 
^ ' > ^ of *a Ho^el B evaluation. ^ , > 

^ . RilC Staff: Oscar iCotJerts 

. „ Session C for Administrators ^ 

Policy Issuer: , T^jpt bias, grade-equivalent 
scores, Evaluating NCE gaips, Siiall • 
^ group discussions on issues raised by 

administrators, c j 

V RMC Staff: Barbara Fagan, Kast tallmadge 

12:00 p.d', ^- 1:15 p.m. Lunch 



1:15 p.m. - 4:30 p.m. General Session _v 

Miscellaneous technical issues. Review 
^ of hazards. Discussion of special 

problems. 

RMC Staff: All 
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The afternoon presentation on the third day became more of a wrap«-up^ 
meeting for the entire group, again with^^^^^llel sessions eliminated* 
First the main features of the system and the three models were Reviewed • 
Suggestions were given, for using the various materials provided •to each 
participant as a resource for further study and training • Staff members 
pointed out which materials would be helpful in connection with-specific 
problems,* such as sampling to, get local norms for use with Model kl* Then 



the session was thrown open for general , questions on an'y aspects of the 
system, or on policy-related mat 



tre^s. 



The low demanfcl for small-group tutorials showed that participants 
sharped certain common concerns and questions much more than they felt a 
need to explore issues peculiar to their own situation. •In some cases, 
however, £^irly sp^ific technical points proved to be stumbling blocks 
for* almost everyone\vand had .to be given more extensive treatment than 
.anticipated. The regression effect error ,^ for example, became the sub- 
ject o-f a separate lesson within the presentation on Model A. 

Tuaining Materials ^ 

On the* first morning of each workshop, all participants , were given 

} 

a set of materials designed to serve as a permanent reference for them, 
and as a resource to.be used in training others in their local areas. 
These materials were contained in a large three-ring binder, so that they 
could be easily removed or re- inserted. In addition to the materials 
provided for ek^ individual, every State received one set of overhead 
transparencies identical to those used by the. KMC staff in ^he presen- 

^"^tations and a* set of four videotapes. 'The following section-^sciMJies^ 

.each of the training materials- in detail. 

User^s Guide . A pocket' in the back of each binder contained a copy 
of the User^s Guide to the^ Title I evaluati^ and reporting system. This 
Guide con^tains the essential information about thq^ system. It presents, 
in decision- tree format, a guide to choosing an evaluation model appro- 
priate to local circumstances' and constraints. It then*^ discusses the re- 
quirements far implementation of each of the models in turn, and provides 
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references to other documents within the system that present even greater 
de-tail on specific topics, for example,. the statistical adj\i8tmeiits that 
may be used for nonequivalent control groups, oj: the selection of local 
norming samples. .The final chapter discusses some additional considera- 
tions, such as coordinating and scheduling activities for .^e evaluation, 
analyzing the data, and the assumptions .made vrtien NCE gains are averaged. 

Forms and Instructions . Each binder 'also contained a complete set 
of forms and instructions. These were set off by divider tabs and arranged 
by levels to correspond to the flow of information in the evaluation and 
reporting system: first the building level, thpn the LEA level, then the 
SEA level. Within each level, the forms were color coded. White was used^ 
for the general information forms ^for reporting on participation, parent^' 
involvement, project cost, personnel, and training. Forms for reporting 
project impact at the building and LEA levels were color toded according 
to which evaluation model _was used. They were pink and salmon for Models 
Ar^and A2, yellow and gold for Models Bl, and B2, and blue and green for 
Models CI and C2. State-level impact forms for presenting the data aggre- 
gated across all projects and evaluation models were again wHite. At each 
^lev^tj^the forms were preceded by a set of instructions explaining in step- 
by-st^p detail how eacl\ page was to be completed. Terms used in a special 
sense were marked with asterisks on the foirms, i'ndicating that they were 
defined in a separate glossary section. 

The forms and instructions were designed to enable each user to col- 
lect, organize, and report afl the impact data required at each level for 
whichever evali^tion model might be selected, as well a^ the general re- 
porting information. Although they can be regarded only as prototypes 
of some future final, forms, they were provided fort all participants to 
use as they might wish, either exactly fS" they were,"^T?!^s a starting 
point for revisions and additions, or in combination with other forms, ^ 
or not at all (some states would plan to handle record keeping and report- 
ing largely by computer). In any case^. State repres^a^atives were in- 
formed that only at the State level would th^re eventually be specific 
requirements "for the type*of data reported, and that at nclevel were any 
particular forms presently a requirement. 
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b'pth the noraed-test 



Problem sets >. ''Problem Siets were developed for 
and non-normed-tes't versions eacFi of the three models (i.e», Models Al^ 
and A2, *B1 and B2, CI and^ C2) . Each problem set was designed as a simu- 
lation of .the actual process of recording ^^d analyzing the data and fill~^*X' 
ing out the forms at the various .reporting levels. fhQs participants - 
working through the problem sets would play the role, first of all, of a 
building- level eval ua tor then of an LEA-level evaluAtor, and Jii\ally-of 
an evaluatox at the State, level. (Later, when the. problem sets were ishbrt- 
ened , some of * them no linger required* all of these levels to be filled in 
by the participants.) , ' . , 

The problem sets presented hypothetical data for groups of studeAts ' 
on yarious tests. Using appropriate norms tables (provided in the mfitte- 
rials), participants had to make score co^ersions, Q^nn^pAite means, and 
calculate the nb-treatment expectation and post-treatment performance 
in the correct way for each model. They could then figure the NCE gains 
for the hypothetical projects, and aggregate these gains using the LEA- 
and SEA-level forms. 

In addition to providing a comprehensive simulation of an actual 
evaluation using a particular ''model , the problem sets were designed to 
provide practic*e with each of the important variations possible with the 
model. For ex'ample, the Model Al problem set required use of publisher's 
norms tables from a standardized test, and also required the use of local 
norms. The local n(j>rms had to be develop'ed by participants, using hypothetical 
data from ^ local norming sample. The Model B problem sets involved the 
use of statistical adjustments for nonequivalent control groups, a prin- 
cipal-axis adjustment with Mo^el Bl, and a covariance adjustment with . 
Model B2. ^ " , 

While participants were doing the. problem se-ts, immediate feedback 

was provided by means of answer keys ^projected on overhead transparencies 

at the completion of each stfep. Any questions that arose about a .par- 

I 

ticular step could be answered on the spot by RMC^taff, who circulated 



thr^oughout the room to offer tutorial assistance. 
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Technical papers ♦ In order to provicie further information on some of 

the more technical points of the system, a series of 14 brief technical 
— 4 

-papers was prepared • Each one dealt witlj a specific topic relevant to 
one or more of the models, and was intended to -serve as a source to which 
Workshop participants, and those whom they would subsequently tifkin, could 
refer for detailed explanations • ' , 

Varying in length ^rom 4 to 15 pages, the technical papers were bound 
as a series of small, attractive booklets, with different-colored covers 

,so that they could be easily distinguished • To facilitate reproduction, 
the^booklets were actually made With 8-1/2 x 11 inch sheets of paper 

^folded in half; when opened out, they could easily be , reproduced by any 
type of^ copying machine • 

As a result of feedback received during the course of the workshops, 
four additional papers were prepared for incorporation into the system. 
All 18 of the' papers ^re described briefly in Table 1, which also speci- 
fies the models to which each of them is applicable, 

Videotapes , To assist SEA personnel in conducting their own train- 
ing workshops, four videotapes were prepared. One of these w^s a 40-minute 
overview of the system, including brief stimmaries of the three models and 
the NCE metric. The three others,' each about 20^inutes in length, dealt 
with the models in somewhat greater depth. The first covered Model *A, the 
norm- referenced model, including both Model Al for tests with national or 
local norms, and Model A2, for non-normed testes, A second tape. dealt With 
control group Models Bl and B2, The principal axis and covariance adjust- 
ments for. nonequivalent control groups were explained, and rules were given 
for converting scores from non-normed tests into NCEs« The last tape was 
on the special regression Models CI and C2, and included discussions of 
how a regression line is derived and how the gain is measured in both the 
regression-discontinuity and regression-projection models. 



. ^Copies available from the Office of Planning, Budgetings, Evalua- 

tion, U,S, Office of ^ucation, 400 Maryland Avenue, S.W,, Washington, D,C,, 
20202, 
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TABLE 1. SUMMARY OF TECHNICAL PAPERS 



Models to which Paper is Applicable 
Norm- Control- . Special' • 



Referenced 



Group 



Regression 



Title bf Paper and Brief Description 



AI 



X5 



Bl B2 



CI C2 



Psychometric and Interpretatibnal Problems 
with Grade-Eq-uivalent Scores - Summarizes 
anomalies inherent in G-E scores and problems 
with their use to measure projedt gains. 

Interpreting NCEs - Explains rationale of 
Normal Curve Equivalents scores and how to 
use them for project evalua^tion and reporting 
gains • 

The Regression Effect - Gives the theoretical 
basis of the regression effect error, and de- 
tails how and why it requires use of separate 
selection and pretest measures in Model A, 
special care in selecting treatment and con- 
trol groups in Model B* 

Selecting a Norm-Referenced Test - Ptesents 
guidelines ^for choosing an appropriate test 
to use in norm-referenced evaluations. 



X • X 



X X 



X X 



X* 



Characteristics of Eight Commonly Used, 
Nationallyg^fTormed Tests - Summarizes what 
forms and levels are available, when testing 
must be done to c6li?€ide with normative data 
points, what score conversions are needed* 

Out^of-Level Testing - Discusses when out- 
of-level testing is appropriate, how to con- 
vert^ out-of-level raw scores to in-leve^ 
percentile scores. - ' ^ 

Local Norms - Explains how local norming 
samples^ can be obtained an(^ tested* 

Types t)f Test Scores - Summarises the dif- 
ferent types of scores and their uses. 

Score Conversions - Gives, instructions 
for using publishers'' norms tables to 
convert raw scores to standard scores, to 
percentiles, etc.' 



ERIC 
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TABLE 1 (continued) 



Models to which Paper is Applicable 



Norm- 
Referenced 



Control- 
Group 



Title of Paper and Brief Description 



AI A2 



<iBI 



Special 
Regression 



B2 CI ^^G2 




Composite Scores - Tells how to combine • 
several measures to get selection/pretest 
scores in Model C 

Criterion-Referenced Tests - Explains 
properties of criterion-referenced and 
norm- referenced tests*, how non-normed 
tests can be used with each of the models. 

Statistical Adjustments for Nonequivalent 
Control Groups - Presents the rationale 
:and instructions for use of the covariance 
alysis and principal-axis adjustments. 

Collecting Achievement Test Data - Out- 
lines essential procedures for proper ad- 
ministration of tests. 

Common Evaluation Hazards - Gives capsule 
summaries of twelve^ methodological mis- 
takes that frequently invalidate results. 



X X 



X X 



X X 



Test Floor and Ceiling Effects - Provides 
guidelines for identifying presence of 
ceiling and floor effects, and choosing 
an appropriate test level. 

Selecting Students for Title I Projects - 
Describes effects of student selection 
procedures on each o-f the evaluation 
models. ^ 




X X 



X X 



X 



Assessing the Adequacy of Normative Data ^ 
Tells how to judge the adequacy of test, 
norms in terms of representativeness and 
freedom from bias. 

Factors that Influence Test Results ^ - 
Discusses sources of spurious results 
in testing, and ways of avoiding them. 



X X 



' X 



Developed .after the workshops to meet expressed needs. 

^3 



These presentations, given by members of the workshop staff, were T 
essentially similar to the detailed sessions on each model given at - 
the workshops*. As in the \^rkshops, diagrams were used to illustrate 
the conceptual basis of the models. Special video techniques allowed 
implementation rules to be superimposed on the screen while staff mem- 
bers \#ere explaining them.* \ © 

Each State r,eceived one set of videotapes, in cassette or reel-to- 
reel format, whichever SEA personnel specified as appropriate for their, 
playback equipment. 

Overhead transparencies . Each State was also provided with one set 
of overhead transparencies identical to those used by RMC staff- throughout 
the workshop's . The transparencies included: 

• Copies of forms to use in showing how th^y were to be filled in.-* 

• Rule^ for implementing each of the evaluation models. 

• Diagrams illustrating such things as "how the no-treatment expec- 
tation is derived from percentile growth curves in Model A, hdw 
the covariance and principal axis adjustments affect Model B, 

or how*the no-treatment expectation is obtained /rem the compari- 
son group regression line in Model C. 

• Answer keys for the problem sets. 

In addition to the set of transparencies given to each State, every person 
attending received, in his materials, binder , a set of paper copies of all 
the transparencies. Participants could use* these copies to follow along 
and take notes during the works^iop presentations, or they could use them to 
generate additional transparencies. 

«^ * 
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i IV. REVISION OF MATERIALS 



■ERJC 



Once the workshops wer^ completed, the second major task under the 
contract was the revision of the training materials, including' the , forms 
and instructions themselves, as well as the technical papers; During 
the sessions, participants had provided RMC staff with considerable feed- 
back on the usefulness of the materials, and these comments and sugges- 
tions were most helpful during* the revision process. 

Comments and Suggestions from Workshop Participants 

The reactions of participants to the elements of the system itself 
were basically similar to those encountered during the visits to SEAs in 
the previdus year. (For a full'account of these visits, the reader is 
referred to Bessey, Rosen, Chiang, & Tallmadge, 1976.) Many SEA repre- 
sentatives were pleased with the possibilities that the system appeared 
to offer for improving thefi: evaluation practices. Of these, some were 
eager to begin working with the models as soon as they could; others were 
somewhat apprehensive -about 'the extent of the changes they would have to 
make in their present evaluation procedures and whether they had the. ex- ^ 
pertlse and resources to accomplish these changes successfully. ' -Both 
groups were anxious to get as much training as possible. Finally, there 
were a few States that viewed the system as having little to of ferr beyond 
what their present evaluation practices could' already do for them. These 
people were interested in learning how -they could adapt or modify the 
system to produce minimum interference with -their own, and to demand the 
least additional effort in order to meet eventual Federal reporting re- 
quirements; , 

' Response to presentations . Regardless of their feeling's about the 
evaluat ion syst em, participants were generally positive in their reactions 
to the manner in which material was presented during the/ three-day Work- 
shops. They were particularly impressed with the materials that/had been 
designed to aid them in their own training efforts.^ While ymo^ felt that 
the workshops had been extremely valuable, however, it wasi also clear that 
few were equipped to absorb all that was covered during the| three days. 
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^ It appealed that a very important function of the sessions,- in fact', was 
to make participants aware of the areas where they would need to seek 
furthei^^elp and to work closely with their Technical Assistance Centers. 

Suggestions for the system * The SEA personnel had a number of sug7 
gestions to offer on how certain aspects of the system* (mainly the report- 
ing of general, rather than impact, information) might be mojdified to 
match more closely their current practices or to make their data collec- 
tion job more manageable. One comment, for example, was that using a full 
time equivalent count of students served would place too great a Micord- 
keeplng burden on projects and would, not produce reliable data. jThe need 
for consistency in the definitions of terms used in ^ the forms foV the 
Title^, I evaluation and reporting system, and other forms SEAs and M^EAs 
are required submit, was also -pointed out. ^ 

There were also a few suggestions for improving some of the impact 
reporting forms* For instance, during the problem-set sessions several 
J participants pointed out that whenever a project* summary form required 
\worksheets to be completed before a certain number could be filled in, 
a brief instruction to that effect should appear on the summary form it- 
self, not just in the insttuctions . In' general , participants seemed to 
^ f eel that the more self-explanatory *the forms could be made^- the better. 
This had ^been a guiding principle of forms development originally, so sudh 
suggestions could be easily incorpotat;ed . ^ * 

Revision "of Forms and Instructions 

The revisions £hat were made in the forms and instructions were 
intended to reflect the mosi*^ up-to-date information and suggestions 
from SEAs about what kind of reporting forms and documentation would 
be most workable for them.- It was recognized tha^ any ''final" forms 
. \^ould be changed still further in the process of government clearance, 
etc., but the revised Vjersions developed after the workshops would rep- 
resent the completion of RMC's developmental effort. There* were two 
main activities involVem in the revision of the forms and instructions: 
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(1) a* terminology review, which led to changes in the terms and defini- 
tions jised in the system; and (2) .discussions of other issues with imr 
pact on how data were to be collectecl, broken down, and reporte'd," These 

activities, are described in the following sections, 

• ' f , 

^' Terminology review . As a number of SEA representatives had pointed - 

out in the workshops, the usage of some terms in the Title I forms and 

instructions varied from the usage in other currently required forms, 

which was derived from the official handbooks published by the Natiot^al 

Center for Educational Statistics (NCES), Washington, D.C. It was decided 

that an attempt should be made to achieve as much consistency as possible 

between the^ forms and instructions and the HCES standards. Copies of the 

relevant NCES handbopks were obtained, and were searched for definitions 

of all terms that appeared in the glossary of the Title I evaluation 

and reporting system, and of all major terms in the forms. The volumes 

included in this review were: 

Harris, Y. Y., & Seibert,. I. N. The State Education^ ^ency (VII). * 
1975. ^ ' 

. Property Accounting (III) (Draft). 1974. 

Putnam, J. F. Student/pupil accounting (V). 1974. 

Putnam, J. F., & Chismore, W. D. Standard terminology for . 
curriculum and instruction in local and State school systems 
(IV) . 1970., 

Roberts, C. T. Staff accounting (IV). 1974. 

Roberts, C. T. , & Lichtenberger j A. R. Financial accounting (II). 
1973. ♦ ' 

Selbert, I. N. Educational technology (X). 1975. • ^ 

¥ox each term from the glossary or the forms, a check was made to - 
see (1) whether there was an official NCES definition* of the term, and 

(2) whether that definition was inconsistent wi,th the defin;Ltion proposed 
in the system. In general, the recommendations for standardizing the Titl 
I terminology to that of NCES were based upon the following guidelines: 

• If a term in. the forms and instructions was not an NCES term, 
no revisions (other than editing) were recommended. 
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• If an* NCES term appeared in the forms and instructions and had 
a definition inconsistent with NCES' usage, a new term was 
recommended • 

• If a term unique to the forms and instructions could be replaced 

— "^th an already existing NCES term and the definitions were con- ' 
# sistent, it was recommended that the NCES term be adopted. 

9- 

• If a similar term appeared in both the NCES* handbook and the 
forms and instructions, an alternate definition might be recom- ^ 
mended to preserve the unique purpose of* the term. 

s 

• If a term ha^ become obsolete because of "a proposed revision, . 
it was r6c ^ ! NMiU ied that the term be deleted. 

Other issues . Several other issues relevant to revision of the 
forms and instructions had arisen during the course of the workshop 
project.* These issues, and their resolution, are discussed in the , 
following sections. 

\. 

1. How to count student participants — Several different break- 
downs of student participation were requested in 'the proposed report- 
ing system. The counts of students included: 

a. The number of students served by Tittle. I a^ each grade and 
each service area (form B-1). . 

b. The number of .students in each project (form B-3). 
t ft 

c. The number of students at each grade level in each project 
(form B-5) . 

These counts were requested at the bulJLding level and at other re- 
porting levels. .There was a variety of ways -to define how these counts 
could be computed. The choice of which definition to u^e depended on both 
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practical considerations and the purposes for which the data were to be 
collected. In addition to providing information about the number of stu- 
dents served at various levels, the participation counts would be used 
to calculate project cost per pupil and to judge the adequacy of the size 
of the^ evaluation. sample . Thus, it was important that the method used 
to compute the participation counts prod,uce data compatible with these 
purposes . 

/ 

The advantages and disadvantages of using each of the following 
definitions were reviewed 



• full-time equivalents ^ 

• average Title I membership 

• average Title I attendance 

• a continuous body count 



It was decided that a body count was not suitable to use in 5eter- 
mining average project cost per pupil or in assessing ' the adequacy of 
the data sample. Furthermote, aggregating body-count data at ^ each re- 
porting level would produce duplicated counts. ) 

Full-time equivalent counts were afi unattractive choice because,, 
as workshop participants had pointed out, they required the, largest 
amount of^record keeping. Also, it was doubtful that any satisfactory 
definition of a full-time-equivalent Title I student could be^ found. 
In additibn, counting the numbei; of students with test data in FTE 
units to judge data representativeness would be conceptually unsatis- 
factory to many system users and would probably result in more errors. 

Computing average attendance 'data would provide very similar parti- 
cipation figures* to those in FTE units, but' would nociequire as much 
record keeping. However, the use pf either, average attendance or FTE 
units to compute costs would produce overestimates of the amount of money 
spent on pupils in a project and^the adequacy of a project's data sample 
was likely to be overestimated when judgments were based on attendance. 

The n^ost promising method for computing paretic ipation counts ap- 
peared to be the use of average Title I membership. Membership counts 
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would be less difficult and time consuming to calculate than attendance 
or FTE counts. Membership figures w^ould be mor^e appropriate to use in 
computi-ng project' cost per student and in deciding the adequacy of the 
size of the data sample. Unlike body-count data, average membership- 
counts when aggregated would not become ^duplicated . For these reasons,^ 
it was concluded that average Title I enrollment would be the most use- 
ful type of participation data to collect in the reporting system. 

2.' What^ if any» cost data should be collected — ^At the time when 
the workshops were conducted, the system called for project cost infor- 
mation on the LEA-level impact forms.' Only a gross breakdown was re- 
quired ,(Title I funds, other supplementary funds, and total), and LEAs 
were not reauired to follow any specific accounting practices to arrive 
at the numbers they provided. Nothing as sophi-sticated as a resource 
cost approach (which would be the only way the figures could be made 
comparable across different parts of <the couittry) was even hinted at. 

It seemed impossible^, within the scope of the system^ to collect 
sophisticated cost statistics. There appeared to be two reasons why the 
collecting of cost information should either be abandoned or postponed 
until, more adequate cbst- accounting procedures could be prescribed, ^e 
first reason was th^t the data provided "^ight be so, crude and error-laden 
as to bear little relatioiiship to the truth. T^he second reason was that 
the c^ata might be over- interpreted , leading to unfair cost-ef f ectiveuess 
comparisons ajnong projects. 

While neither 'of these possibilities could be denied, it was felt 
that there would be .a substantial correlation between the cost data 
actually reported^ and those that would be reported under an ideal ^sys- 
tem. Thus the data would have some utility. Some overinterpr etation 
,of the data would be likely, regardless ^of all the cautio,ns that might 
be offered. It was concluded, h'otfever, that the benefits to* be gained 
from comparisons that did not go beyond the precision of the data could 
outweigh the negative impacts of comparisons that did exceed those 
limits. * _ 
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3. What ^ubi^ct breakdown to use for impact data — Originally the - 
system reque.sted information on two types of cognitive skills — verbal and 
numerical. This breakdown was' used both for reporting pupil participa- 

' tion by piroject area and for reporting impact. While" the categories ver- 
bal and numerical were considered adequate for administrative purposes 
such as reporting pupil participation, t.here were several alternatives 
that sOme felt offered a more useful breakdown of information .for report- 
ing impact. It was pointed out, .for example, that the terminology used 
for impact reporting purposes should be meaningful and appropriate in terms 
of cimriculum and of measureji^ent . For this reason, a change from "numer- 

^ i6al" to the ^ore familiar "mathematics" was recommended. In the verbal 
area there was some question as to what sort of breakdown should be used. 
"Verbal" covered several more or less clearly defined subject; areas, which 
could be specified individually. The disadvantage of "verbal" was that 
it did not isolate certain subcat^Dries that were clearly identifiable 
and Vere of fnterest to decision makers, t^ie most Important of these being 
read ing • 

There appeared to be no reason why reading could not be successfully • 
broken out as a separate c^teg(fry. It was unambiguously defined both as 
a curriculum are^and as a measurement area.> Since so much effort within 
Title I focused upon the improvement of reading skills, it woulth^e use- 
ful to know how many pupils were receiving services in this area. Fur- 
thermor^e, when achievement test data- were reported, a separate category 
for reading would mean that the results aggregated there were all derived 
from reading tej^ts and represented the outcomes of instruction in read- 
ing. More meaningful interpretations could be made of the information 
in the "Achievement Gains by Project Characteristics" matrix on form S-5, 
because comparisons could be clearly specified as inv|Hring reading proj- 
^ects. only. Therefore it was decided that Impact data should be catego- 
rized into^ "Reading," "Language Arts Other than Reading," and "Mathematics." 

4. Whether to aggregate project impact data by grade. level or across 
grade*" levels — Impact da^a were to be aggregated at the State level on two 
forms, S-4 and S-5. One would provide achievement data by grade level 
across the' State; the other would give statewide achievement gains by 
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project characteristics • The question of how impact data should be aggre- 
gated involved mainly the gains-by-pr o^ct-characteristics matrix on S-5. 
There were two possibilities: 
>.-^^ 

Option 1 — ^Aggregating gains by project characteristics across 
grade levels • 

Option 2 — ^Aggregating gains by project characteristics sep- 
arately for each grad^ level > 

Requiring both aggregations was not considered because of the burden it 
would place upon State personnel, who would aimply have to break down the 
same data in two different ways. 

Aggregating ^aihs by project characteristics across grade levels was 
the procedure originally used in the system. On form S-5, NCE gains, 
broken down into six ranges, would be Teported for four types of project 
characteristics: hours of instruction per week, total hours of instruc- 
tion, instructor-to-student ratio, and cost per student. Requiring that 
gains be aggregated separately for each grade level would have meant that 
States would fill out form S-5 once for each grade level at which achieve-- - 
ment data were reported'. This would entail as many as 12 or 13 S-5 ^orms 
for each State, depending on how many grade levels included ' cognitively 
oriented Title*^ activities. Although this would obviously mean more work 
for State personnel, it would also provide more information. In decid- 
ing which procedure to require within th'e system, it was necessary to con- 
sider whether the more detailed breakdown would have enough usefulness 
to decision makers to be worth the extra effort. 

The most critical requirement was that the g-^ns reported be as re- 
liable as possible. Small sample sizes could threaten the inter pre tab il- 
ityof gains from many Title I pi;ojects aa it was; breaking results down 
by grade level would simply exacerbate this problem. In many cases, the 
^ains reported would represent fewer than 30 children, making the likeli- 
hood much greater that random errors would distort the true picture of 
a project's grade-level impact. On the other hand*, gains reported for • 
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I an entire project would represent the maximum number of children, allow- 
ing greater confidence that the figure for. each project was not distorted 
y random errors, but was an accurate reflection of the instriy^tional 
treatment's true impact. It'was therefore recommended that aggregation 
of gains by project across grade levels be continued. 

5. Whether to add a summary of total-project data — The possibility 
of using "project Vectors" represented an altogether new addition to the 
system. A project vector might look something like the following: 

#4328 GR 4-6 LA 70 S/G H/W 2.5 T:H 75 I/4S $285 3NCE 

This would translate tq^: "Project Number 4328 provides 'language arts 
other than reading instruction to students in grades 4 through 6 with 
an average enrollment of 70 students per grade . --The, treatment is 2.5 
hours per week for a total of; 75 hours per year. There- is one instruc- 
^^^^^r for every 4 students and per-pupil cost is $285. The prdject pro- 



^i)ced a gain- of 3 NCEs." 

Such vectors would be quite simple for a cQmputer to produce. The 
task would place a considerable clerical burden on SEAs that did not 
generate their report data b^^^omputer . However, with minimum redesign 
of the LEA reporting form L-13, all the vector information could be ar- 
ranged in an easily tr anscribable format. 

Since the State reporting forms S-4 and S-5 provided ho total- 
project data, it was recommended that project vectbrs be incorporated 
" into the system. 'This resulted in the addition of a nev/ reporting form'* 
^ at the JState level, form S-6. - 

. Technical Papers 

, The 14 technical papers that were produced as' part of the training , 
materials for the workshops were described In Chapter III of this report* 
After the workshops had been completed; these technical papers were re- ^ 
vised. In addition, four new technical papers were produced, making a 
tal' 6f 18. 
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Revised technical jlapers ♦ New, revised versions were produced for 



the 14 original technical papers • Revisions were based on comments and 
suggestions from persons within USOE, from Technical Assistance Center 
staff, and from tho^p^who attended the workshops. Some of the technical 
papers had only minor eJltorial changes; in^ other cases^ major revisions 
or additions of new material were made.^ ^ ■ . ^ 

New technical papers * Four new technical papers were also added 
to the system. These dealt with subjects that, from obseryations-of-^ 
those attending the workshops, appeared to need more extetisive treatmeiit 
in the system docuipntation. ' 

A new paper entitled Test Floor and Ceiling Effects deals with the 
problems cause4 by these ef,fects and how to- avoid them. Several methods 
are discussed for ^detecting the interference of the floor and ceiling 
when analyzing a group's test scores, and tabled appended ^to the paper 
show percentile levels at which floor and ceiling effects occur on eiglh: 
commonly used, nationally normed tests. The principles discijssed apply 
to all tes'ts,' however. This paper is intended Co help evaluators in 
choosing a test level that will be appropriate for their Title I group, 
and should l^e useful in>connection with all of the models. 

A second paper. Selecting Students for Title I Projects ^ explains 
how the implementation rules for each of the three models affect the 
method of selecting Title I participants. It ^discusses the practical 
implications of model-dictated restrictions — for example7\the fact that 
the pretest cannot be used to select treatment group participants in 
Model A, but must be so used in Model C. 

Assessing, the Adequacy of Normative Data discusses aspectsVof the 
norming prpcS^dures that should be examined when selecting a oOTm-ref erenced 
test, ih order to judge the adequacy of Ifhje^norms. Among- iThe topics in- 
cluded are freedom from bias in the norming sample, size qf the.sampl'e, 
and some Qf the sources of problems faced by test producers trying to ob- 
tain unbiased^ norms. The considerations outlined in this paper apply in , 
judging the adequacy of either nationally or locally developed tests. 
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The fourlh paper. Factors that Influence Test Results , examinfea^ some 
of the things that contribute to a student's test score. It considers 
the role of actual knowledge and of test wiseness. Guessing and other 
test-taking strategies are. discussed, as well as situational factors and 
administrative procedures that can exert spurious influei^pes on scores. 
Recommendations are made on' how to avoid these problems or how to mini- 
mize their biasing influences. 

Revision of the User's Guide 
' , J 

The User's^Guide to the system was also revised. ' In addition to ' 
minor changes in the various chapters, a new chapter entitled "Trouble- 
shooting and "Refinements" was produced. This chapter discusses some addi- 
tional analyses that local evaluators may wish to undertake in adc^iti^n 
to following the requirements of implementing the particular model chosen 
It also discusses steps that may be taken to "salvage" evaluations that 
have been incorrectly implemented or y.awed in some other way (e.g., test 
.ceilings or f loots are encountered). 

Some new appendices were also added, and the tables for converting 
-percentiles to NCEs and vice versa were revised to show all numbers to' 
two significant figures. This meant that percentiles and NCEs from l.to 
9 were carried ^ut to one decimal place, while all ojzhers wete shown as 
whole numbers. 
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V. CONCLUSIONS AND RECOMMENDATION'S 

Training Problems and Issues * ' ^ 

Durfng the course of the workshops, much was learned about the prob- 
lems and the complexities of running an intensive training program^n a 
nationwide scale • Some of this &cperience is summarized in the following 
sections • ^ ^ 

Diversity of participants^ expertise * The major problem facing the 
workshop staff was how to deal withj a range of different l«evels of exper- 
tise and different interests • Partyof the problem here was that the^ amount 
^of knowledge each participant would bring to the workshops was not known 
beforehand. All that was known was that there would ^te a great variety. ^ 
In the face of this situation, it was decided that the best approach would 
be to plan the opening presentations for persons with little- knowledge 
of the models and little background in the technical aspects of evaluation. 
For siich persons, the sessions should be understandable and helpful, while 
for those with more sophisticated training they should at least provide 
a useful summary of the system. ^ 

' Once the overviews were finished, the plan was to individualize the * 
curriculum by dividing the groups into parallel sessions for administra- 
tors and evaluators, and by further subdividing thes.e sessions into small- 
group tutorials. The first division was definitely essential; interest- 
ingly enough, the second division did not appear to be. As discussed in 
Chap-ter IH, these small-group tutorials were often not actually used; 
rather, the administrator and evaluator sessions seemed to be suf f iciently ^ 
indiv idolized as they were. This was perhaps due partly to. the fait that 
two or three RMfc staff members were always present at each parallel ses- 
sion. ^Thus , for example, if a problem set was being completed in- an eval- 
u^tor session, several staff members could circulate among the trainees 
to answer individual* questions, even as one staff member wa^s taking the 
whole group through tl>e problems using overhead transparencies to display 
the answers • , 
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It seems likely that the reason why the small-group tutorialsVwere 
not as useful as anticipated was that, despite the variety in the iri train- 
ing and interests, the participants were practically all approaching the 
system as a new development. Many of the SEA personnel had been briefly 
introduced to the system during the site visits a year before. However, - 
few actually had experience using tlie models, or even in formulating def- 
inite plans for implementation^, Undoubtedly if those attending had been 
further along-in their own. use of the system, there would have been more 
demand for smail-group 4^essioris focusing on particular local- problems and. 
issues". '^As it was, the evaluators and, to an even greater extent, the 
administrators, seemed to share a number of common concerns, which could 
be most satisfactorily dealt with. in the single sessions designed for each 
group. ' V 

. Allocation of time . .Another question about the training strategy 
involved the allocation of time. How much time was to be spent on each 
of the technical points was determined basically by their relative impor- 
tance and anticipate^' usefulness 'to participants. Thus, for example, 
more time was spent on Model Al, which was likely to be the most widely 
used model. A second and more difficult aspect of 'this question involved 
how to allocate time between presentations by workshop staff and questions 
or comments from the audience.. Because of the large amount of material 
to be coveiDed, most of the "time was originally assigned to presentations. 
Allowance was always made, however, for questions to be asked at the end 
of each presentation. Since the technical material was generally new to 
^most participants, staff members felt that they had been successful if 

there* were a few technical questions, but not. too maay. While some ques- 
. tiojis could indicate that trainees had acquired a ^ood basic grasp of the 
subject, a large number of questions, particularly elementary questions, 
signaled that the presentation had failed to give the listeners an ade- • 
quate understanding of the material. 

As -the workshops' progressed, it became apparent that participants 
favored haying a large general session open to questions at the end of ^ 
the last day. The original plan-for that afternoon was to have small- 
" group tutorials on issues of interest to various groups of participants. 
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By the last day, however, the groups seemed to want to meet together 
and to air their various questions and comments ♦ The tutorial sessions ' 
were therefore dropped in order to provide a whole-group question-and- 
answer session* - ' * . 

V 

• Miscella neous training problems * One fact that became clear during 
the workshops was that the amount of preparation required should* not be' 
underestimated. Staff members spent many days writing technical papers; 
devising ptoblem sets; designing graphs, drawings, and diagrams for the 
overhead transparencies; and preparing their presenta'tions for the video-' 
tapes and for the workshops themselves. Needless to s^y, the presenta- 
tions improved with practice at eac»h successive session. Another aspect 
of the' preparation effort was the arrangement of all necessary support 
services. In. addition to the efforts of the traveling workshop "staff , 
one *full-time coordinator was required to handle 'tjie hotel reservations, 
airline tickets, reimbursements, and the printing, packings and shipping 
of materials. » 

Another factor that should not be underestimated * is the importance 
of good, facilities. 'The best efforts to produce top-notcii materials and 
polished presentations can be negated if workshop participants cannot see 
or he^r them easily. In some cases, difficulties were encountered in 
getting screens, chalk or erasers, etc. There were some rooms where the 
heating or air conditioning made so much noise that a microphone had to 
be used — w inconvenience when one is also moving around, using a point- 
er, or trying to change overhead transparencies.. Despite careful attempts 
to ensure adequate facilities in advance, meeting rooms vere not always 
ideal. At' one workshop, the agreed-upon conference room was apparently- 
preempted by a larger group, and the hotel offered a/substitute room" that 
was too small. Fortunately, the session was able to be moved to confer- 
ence facilities kindly offered by the regional Technical Assistance Cen- 
ter. In another case,^ the meeting roqps were so cold that people sat in 
'^thei? coats, as the heating system tried vainly to cope with unusually 
harsh winter weather. 

The best-way to avoid unpleasant surprises with the conference facile 
ities is to inspect them in ^ierson or to choose hotels that are already 

4 . ^ / 52 

58 



known to staff memters • Ins three of the ten workshop locations^ this was 
posa^ible, and the facilities were enMrely satisfactory. If this is not 
possible, it is a good idea to get asinuch detailed infonna|:ion as pos- 
sible about the meetin^rooms. For example^ actual room dimensions' should 
be jrequestei, rather than just the hotel's estimate of how many people 
the room will seat • " ' ' • 

Materials and equipment ar^ as important as facilities. Arrange- 
ments for microphone, podium, screen, vblackboard, etc., should be clear- 
^ly confirmed. Among the materials used in presenting these workshops, 
the oveVhel^d projector was one of the most valuable. The transparencies 
wereAlarge, bright, and easy to see. They appeared to be extremely^usef ul 
in helping participants visualize and remember the concepts underlying 
the models, and to undeAtapd practical examples. Another advantage was 
that/ the i^lustr'ations\^ technical material could be carefully made up 
ahead of time, rather thaiTiiastily drawn on a chalkboard while a presen- 
tation was in progress. When impromptu illustrations were needed in the 
midst of a presentation or in response to ar question, special pens could 
be used to draw or writ^ on blank transparencies. Chalkboards were some- 
times* used as a supplement to the overhead transparencies, but they were 
found to be less effective — too small, poorly lighted and in general more 
difficult to see. Another useful aid was the set of hard copies of the 
transparencies that was included in each person's binder. 

Cost of Training ► ^ ' 

The cost of providing three days of workshop training for represen- 
tatives of all SEAs is summarized in Table 2. The figures for materials 
provided to each State are kept' separate from those for items provided 

>^o each participant. The former materials were, in fact, intended for 
use by State personnel in conducting their own training efforts, rather 

. than for use as a part the workshops. . 

As can be seen from Table 2, the combined cost of travel and per 
diem expenses was the largest part of the expense for individual parti- 
cipants, averaging nearly $212 per person. Materials costs averaged 
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TABLE 2. SUMMARY OF WORKSHOP COSTS 



1 


Total Costs 


Unit Costs 












MATERIALS (600 set v 








Binders ' f ' 


^ 1,480 






i.naex axviaeLb 








Sheet protectors 


259 






Buttons 


116 






Forms and Instructions 


3,225 






Problem sets 


1,167 " 






. Technical Papers 


5,725 






Transparencies (hard copies) 


936 


* 




User's Guide 


1,565 






Totad i 


$15,060 


$ 25aO/set . 





TRAVEL (for 167 participants reimbursed by RMC) 
^ Air ^ 17,465 

Local * ' ^ ^ 3,221 



Total 



$20,686 



$123^86/participant 



PER^DIEM (fc/r 167 participants reimbursed by KMC) 
Meals /Lodging 1^ $14,705 



$ 88^05/participant 



MISCELLANEOUS (for 300 workshops attendees) 



Telephone 
Postage 
Staff Mileage 
. Xerox 
Supplies 
Screen rental 
Hotel facilities rental 
Refreshments (coffee break) 

Total 



1,016 
980 
212 
557 
391 
24 
500 

1,890 

$ 5^570 



$ 18^57yattendee 



STATE COSTS (57 sets) * 
Transparenc ies 
Videotapes 



1,343 
10,492 



Total . • q>ii,0J3 q>zu/ ♦Dj/ state 


GRAND TOTAL ^ 


'w $67,856 


> 




♦ 

GO 





•4 



/ 



-ft? - 



$25 per. person, and miscellaneous costs another $19» Extra materials 
were produced, allowing each State to be given additional sets for use 
in their own train4.ng» ■ ' . > ^ 

Costs "not "shown in Table 2 were basically the salaries and travel 
expenses of the^ five-persons on the workshop instructional team, one 
full-time scheduling and materials coordinator, an(i office pupport staff. 
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APPENDIX 



Alabama 

Ro,d Hildreth • 
David Nettles 
Edward Spears 

Alaska 

Ruth Harris 
Steven Hole 
Lawrence Schutt 

Arizona 

Donald Kearns 
Gary Mossman 

Arkansas 



ESEA TITLE I EVALUATION WORKSHOPS 

, ^ * 1976 - 1977 

SEA Participant List 

Delaware 

Atwood Badman^ ' 
William Corkle 
Edwin Skinner 
Janet Wall 

District of Columbia 

June Bland 

Jan Dell Conley 

Earl Hunter 



Andrew \ 
:t Kerr \ 



Dean 
Robert 
Clarence Morrlis 



California 

Edward Bispo 
Manual Cej a 
James Fulten 
Claude Hansen 
Jane, Vinson 

Colorado 

.Robert Cheuvront 
Terry Lawson 
Barry Shaffer 
Charles Shaffer 

Connecticut 

Alice Bordonaro 
Ernestine Brown 
James Burke 
Shirley Foster 
Patrick Proctor 
Edward Ricciuti ^ 



lorida 



Robert Friedman 
Halley B. Lewis 
Rolland Mielke 
Juanita Parks 
Clyde Stevens 
James Temple ^ 

Georgia 

R. C. Beemon 
Sarah Moore 
William Tidwell 
Susan Underwood 
Ann White 

Hawaii 

Pao-Ming Tchou 
Rose Yamada 
Harold Wilfong 

Idaho 

Michael Brunner 
Donald Carpenter 
Ruth Sydell 
John^Taggart' 
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Illinois 

Norman Stenzel 
'Paul Taubr 
• Connie Wise 

Indiana 

John Heseman 
Barbara Pashos 
William Strange 
Fausto Vergara 

Iowa 

Gilbert Hewett 
Oliver Himley 
Ronald Huff 

Kansas 

Jayanne Angel 1 
Kenneth Gentry 
Donald Hardesty 
Ann Harrison 
William Lange 

Kentucky 

Jacqueline Cantrell 
William Field 
Donald Hart * 
Donald Van Fleet 

Louisiana 

Virginia Gerace 
Elizabeth Hensley 
Charles Jarreau 
Richard Owen 

Maine 

Donald Christie 
Charles .Cosgrove 
Donald Graham 
Donald Zambri \ 
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1 <^ont*^mioHi 


•* 


Maryland 


Nebraska 


North Dakota 


Eugene Adcock 


Elizabeth Alfred 


Warren Borchert 


Charle^ Burns 


Jack Baiilie 


Alton Koppang 


XSeorge -Lisby 


Douglas Colberg 


David Lee 


Lisa Spurrier 


Ervin DeBoer 


Minard McCrea 


Joan Stevenson 


Jess Medina 


Ronald Torgeson 


Percy Williams, 


Mert Smith 






Ohio 


Massachusetts 


Nevada 








Arlie Cox 


Jack Baptist^ 


Richard Gunkei 


Carl Evans 


Robert Consalvo 


R. H. Mathers 


Ken Taylor 


John Howell 


Harold Sayler 


* 


Robert Reid 




Oklahoma 


Richa^rd Zusman 


New Hampshire 






Jame^ Casey 




> Michael Brophy 


Frank Hobbs 




Ma C KHiglt t===^Dia^^KJ_^ 


fciciwara nuey 


David Donovan 


Richard, Hodges 


^Roh^ t Maxwell 


Eugene Paslov 






Daniel Schooley 


New Jersey 


Oregon V \ ^^^^^--^ 


Minnesota 


Thomas Corcoran* 


Gordon Ascher 




Sherwo#d Gordon 


Fred Buehling 


Clyde Bezanson 


Eugene Mason ^ 


Robert Clemmer 




Joseph Moore 


Jerry Fuller 


Mississippi 




Mark Greene 




■ New Mexico 


Marshall Herron 


A. £. Bilbo 


< 


Barbara Hunt 


Albert Comfort 


Livie Duran 




Frank Drummonds 


Charl4s ^Epler 


Pennsylvania 


R. B. DuBoise 


Gilbert J}artinez 




W. L. Herd 


Paul Rost 


Frank Reardon 


Samuel Parker 


Carol feoss 


Thomas Schur^z 




^ Kerry Wenas. 


Greg Shannon 


Missouri 




* 




New York 


Rhode Island ^ 


Charles Blackman 




\ 


Gary Brummitt 


George Cronk 


^Gini Bilotti 


John Jones 


William Flannigan 


Mary Lynne Bourque 


Amos Morris 


Jack House 


Henry D'Aloisio 


Ruth Pair 


Paul Hughes 


Pat'^'DeVito 


Donald Snyder, 




Gladys Thomas 




North Carolina 


Phillip Zarlengo 


Montana 








John Bolton 


South Carolina 


Daniel Ferriter 


William Brown 




Dean Lindahl 


H. T. Conner 


Garlan Hicks 


Jay McCallum 


Gerald Donnely 


Charles Statler 


P/^T^o>"t* Piit*V4 Omo^ro ^ 
IvUUcLL. IVULlldUCyCL 




Lane Tranthen 


Gerald Shanley 


Weaver Rogers 


• * 


• 


Harold Webb 
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SEA Participant List (continued) 



South Dakota 

Clint Berndt 
Dennis Gibbs 
Robert Travis 

Tennessee 



Joleta Reynolds 
Chris Satterfield' 
William Tom 1 ins on 

Texas 



Frank Contreras 
Richard Hardebeck 
Joseph Lopez 
Or en Poage 
Mary.in Veselka 

Utah 

Jay Donaldson 
David Nelson 

Vermont 

Gerry Asselin^^ 
Marshall Knight 



Virginia 



L. Bruce Johnson 
W. H. McCann • 
W. E. Newell 



West Virginia ^ 

Charles Duffy 
Edward Moran 
David Purdy 
Robert Taylor 
James Thom'pso n 
Philip Thorn 

Wisconsin 



on I 
ton^ 



Clem'taime 
Frank Burkholder 
Frank Evans 
Gail Krc 

■ ^ 

Wyoming 

George Bohl 
Jerry Lewis 
Kathleen Verville 

Ame ric ^Rjj ^S amoa 



Sili Atuatasi 
Carol Golanbeski 
L» " Tagoilelago 

Guam 

• Josefph Cruz 
^thony Kallingal ' 
Lillian Lujan 

Puerto Rico 



Trust Territory ^ 
of the Pacific 

Edward Klingbefg 

Birch Robison 

Damian Sohl * 

Virgin Islands * 

Austin Donovan 

Janet Griffeth 

Agatha Jarvis , 

Kurt Komives 

Bureau of Indian A^airs 

Verli«i Belgarie 
Jan Dreibilbis 
Sandra .Fox 
^Robert Hall 
Gene Knight 
William tong 
Noel Mal(ix;ie 
Paul Mel choir 
Heinz Meyer 
Stewart Munz 
* Tom Patterson 
Gilbert Rogers 
Neil Reece 
Annabell ^Rosenbluth 
Earl YeaWquo - 



Washington 

Marion Cupp 
\^illiam Hulten 
Daniel Organ 
John Schlotsfeldt 



Marta Barros 
Blanca Cacho 
Fanny Freytes ^ 
Domingo Or,tiz 
Aida Rodriguez* 
Vidal Velez Serras 



JjbTE: Each SEA was invited to send up to three participants whose ex- 
penses would be covered by KMC under 'the terms of the contract. 
Add±^tional participahts att-ended at their own expense*. 
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